Mining software repositories provides developers and researchers achance to learn from previous development activities and apply thatknowledge to the future. Ultra-large-scale open source repositories(e.g., SourceForge with 350,000+ projects, GitHub with 250,000+projects, and Google Code with 250,000+ projects) provide an extremelylarge corpus to perform such mining tasks on. This large corpus allowsresearchers the opportunity to test new mining techniques andempirically validate new approaches on real-world data. However, thebarrier to entry is often extremely high. Researchers interested inmining must know a large number of techniques, languages, tools, etc,each of which is often complex. Additionally, performing mining atthe scale proposed above adds additional complexity and often isdifficult to achieve.The Boa language and infrastructure was developed to solve theseproblems. We provide users a domain-specific language tailored forsoftware repository mining and allow them to submit queries via ourweb-based interface. These queries are then automaticallyparallelized and executed on a cluster, analyzing a dataset containingalmost 700,000 projects, history information from millions ofrevisions, millions of Java source files, and billions of AST nodes.The language also provides an easy to comprehend visitor syntax toease writing source code mining queries. The underlyinginfrastructure contains several optimizations, including queryoptimizations to make single queries faster as well as a fusionoptimization to group queries from multiple users into a single query.The latter optimization is important as Boa is intended to be ashared, community resource. Finally, we show the potential benefit ofBoa to the community by reproducing a previously published casestudy and performing a new case study on the adoption of Java languagefeatures.
展开▼
